.NET RESTful Engine Performance and Best Practices

This article outlines some .NET RESTful Engine load tests that we ran, as well as some best practices for hosting your own .NET RESTful Engine.

Performance Tests

note

Please note that pre version 22.3, we used a simple template for our tests. Since then we have decided to use a more complex template that would more accurately reflect average usage of the engine. To account for this, we ran tests for both templates for engine versions 22.3 and 22.4. Going forward (ie. version 22.5+) we will only be testing with the newer template.

For these tests, we used two different "sized" virtual machines on AWS. These are the specifications of the VMs we used:

	t2.small	t2.large
vCPU	1	2
CPU Credits/ hour	12	36
Mem (GiB)	2	8
Storage	EBS-Only	EBS-Only
Network Performance	Low - Moderate	Moderate

The tests consisted of sending about 16 requests per second for 1 minutes (~1000 requests). We measured the time from for first request to the time of the final document becoming ready on the disk. So the time it takes to post and process 1000 requests. Here are the results we found after running the tests.

Performance Test Results

Version 22.4.*

Old Template (Simple Template)

	t2.small	t2.large
Run 1 (Document/ second)	1.71	3.22
Run 2 (Document/ second)	1.62	3.06
Run 3 (Document/ second)	1.96	3.17
Average (Document/ second)	1.77	3.15

New Template (More Advanced Template)

	t2.small	t2.large
Run 1 (Document/ second)	1.49	2.42
Run 2 (Document/ second)	1.32	2.17
Run 3 (Document/ second)	1.41	2.93
Average (Document/ second)	1.41	2.51

Version 22.3.*

Old Template (Simple Template)

	t2.small	t2.large
Run 1 (Document/ second)	1.83	3.45
Run 2 (Document/ second)	1.87	3.13
Run 3 (Document/ second)	1.60	3.67
Average (Document/ second)	1.77	3.42

New Template (More Advanced Template)

	t2.small	t2.large
Run 1 (Document/ second)	1.11	2.08
Run 2 (Document/ second)	1.69	2.63
Run 3 (Document/ second)	1.32	2.14
Average (Document/ second)	1.374	2.284

Pre Version 22.3.*

	t2.small	t2.large
Run 1 (Document/ second)	1.77	2.96
Run 2 (Document/ second)	1.32	3.80
Run 3 (Document/ second)	2.26	3.12
Average (Document/ second)	1.78	3.293

As you can see, the larger VM was able to process documents almost as twice as fast. This is a result of having double the processing cores in the large VM relative to the small VM. Through our testing we found that memory had little affect on performance, and the number of CPU cores was a lot more impactful.

note

Please note that the template used for testing was hosted on AWS S3. If you are running a templated saved locally you can expect performance numbers to go up.

Best Practices

Through our testing we found a couple of "best practices" for hosting the .NET RESTful Engine, especially if you are anticipating a heavy load on your installation:

1. Processing power matters
   • The number of CPU cores available have a direct impact on the document processing of the RESTful engine. The more CPU cores available, the better the engine will churn through documents.
2. Long Initialization Time
   • If you realize that the service initialization takes too long and notices that the first requests take longer than the subsequent requests, you may need to make some configurations to your IIS install. Please follow the steps in the following link
3. Default IIS App Pool Idle Time
   • IIS App Pools have a property called "Idle Time". This property is set to 20 minutes by default. If you are sending huge loads to the RESTful engine in a short period of time, the App Pool will go to sleep in 20 minutes even if the engine is still processing documents in the background. You can "wake" up the engine again by hitting any of its endpoints, and it will pick up where it left off. To avoid this, we recommend changing the app pool Idle Time property . Here are the steps for doing so:
     1. Click on "Application Pools" in the left pane on IIS
     2. Right click on the ".NET v4.5" application pool (your RESTful Engine should be registered under that one)
     3. Click on "Advanced Settings"
     4. In the "Process Model" section you will fine the "Idle Time-out" property, represented in minutes
     5. Change this property to a time period that would work well for you

note

Please keep in mind that even if the engine is processing documents in the background, the App Pool will still shutdown because IIS describes "A worker process is idle if it is not processing requests and no new requests are received". So if you anticipate a large load please update the property described above.